Monocular computer vision aided road vehicle driving for safety

ABSTRACT

The present invention is a monocular computer vision technique. The technique is used to aid road vehicle driving for safety. An camera (e.g., camera or TV camera or camcorder) is installed in a vehicle facing to the road in front of the vehicle to capture a sequence of road/scene images for analyzing the driving situation. The situation includes the distance to a front obstacle or vehicle, the speed of the vehicle, and the left/right location of the vehicle in a road lane.

FIELD OF THE INVENTION

[0001] The present invention is a monocular computer vision technique.The technique is used to aid road vehicle driving for safety. An camera(e.g., camera or camcorder) is installed in a vehicle facing to the roadin front of the vehicle to capture a sequence of road/scene images foranalyzing the driving situation. The situation includes the distance toa front obstacle or vehicle, the speed of the vehicle, and theleft/right location of the vehicle in a road lane. If a danger situationis arisen, the sound and/or light alarm will arise to warn the driverfor safety.

BACKGOUND OF THE INVENTION

[0002] Safe traffic is important to the property of an individual andthe stability of human society. Driving a car out to work or travel istrivial nowadays. Sometimes, persons lose their attention during drivinga car; especially, for a long-distance or a high-speed driving, manydanger situations are then arisen. If there is a mechanism to alarm thedriver and provide some useful information to judge by the driver inthese situations, the danger is therefore avoided.

[0003] Several techniques have been proposed for the purposes of safedriving. In most previous proposed techniques, an ultrasonic rangefinder or laser radar was equipped on a car to detect the obstacles infront of the car. The equipments are expensive, have only a specialfunction, and are not easily used. Other techniques had equipped a pairof cameras on a car and then utilized the stereo vision method to guidethe navigation; however, this vision system is still complicated,working slowly, and expensive. If we can only equip a camera to detectthe obstacles in front of the car and to guide the navigation, thevision system will be simpler, cheaper, and easy to practice for safedriving.

[0004] The left/right location of a vehicle in a road lane is alsoimportant to the driver during driving on a road; especially for thatthe driver can't concentrate his/her attention on the driving. To thedetection of the left/right location in a lane, someone had proposed amethod by equipping one camera on each side of the car to detect lineson both sides of the current lane; however, such equipments are stillcomplicated and more expensive. It had better detect the left/rightlocation of a vehicle in a road lane only using a camera.

THE PURPOSES OF THE INVENTATION

[0005] The present invention is used for aiding road vehicle driving toimprove the driving safety, to reduce the traffic accident, and to avoidthe loss of lives and property. The invention technique only installs acamera in a vehicle facing to the front of the vehicle to capture asequence of road/scene images, then utilizes the monocular computervision method to acquire the driving situation. The situation includesthe distance to a front obstacle or vehicle, the speed of the vehicle,and the left/right location of the vehicle in a road lane.

BRIEF EXPLANATION

[0006] In order to attach the mentioned purposes, the present inventionproposes a monocular computer vision technique to aid road vehicledriving for safety. The invention technique only installs a camera in avehicle facing to the front of the vehicle and then processes thefollowing steps:

[0007] i. using the camera to capture a sequence of road/scene images;

[0008] ii. processing and analyzing the sequence of images using themonocular computer vision method; and

[0009] iii. acquiring the driving situation for the vehicle based on theanalysis results of the computer vision method.

[0010] As the described monocular computer vision technique for aidingroad vehicle driving, the camera is a camera, TV camera, digital camera,camcorder, or digital camcorder. The capture device connects a computerand directly transmits the captured images or video into the computerfor computer vision analysis.

[0011] As the described monocular computer vision technique for aidingroad vehicle driving, the vehicle is a car, bus, or truck.

[0012] As the described monocular computer vision technique for aidingroad vehicle driving, the camera is fixed on the front of a vehicle, thedevice is facing to the road in front of the vehicle, and the contentsof the images are the road and scene in front of the vehicle.

[0013] As the described monocular computer vision technique for aidingroad vehicle driving, the procedure for the monocular computer visionmethod contains:

[0014] i. detecting the distance to a front obstacle or vehicle byanalyzing the image sequence;

[0015] ii. estimating the vehicle speed based on a continuous imagesub-sequence of the sequence; and

[0016] iii. determining the left/right location of the vehicle in thecurrent lane.

[0017] As the described monocular computer vision technique for aidingroad vehicle driving, the detection of the distance to a front obstacleor vehicle consists of the following steps:

[0018] i. from each image to extract the lines on both sides of thecurrent lane and then find the intersection of these two lines; theintersection point is just the vanishing point of the lane;

[0019] ii. from the vanishing point and the known focal length of thecamera lens to find the pitch and yaw angles of the lane lines withrespect to the camera coordinate system;

[0020] iii. from the pitch angle and the height of the camera tocalculate the distance from the camera location to the point which isthe intersection of the camera optical axis and the road plane;

[0021] iv. from each image to find horizontal lines to indicate theintersections of the rear wheels of front vehicles and the road, selecta nearest horizontal line that overlaps the current lane, and judge thatthe horizontal line is located above or below the image center;

[0022] v. from the known camera focal length, camera height, pitchangle, and the vertical distance from the horizontal line to the imagecenter to find the road distance from the camera location to the rearwheel of the front vehicle or obstacle.

[0023] As the described monocular computer vision technique for aidingroad vehicle driving, the vehicle speed is computed based on thedetection of the distance to a front vehicle. At first, a terminal pointof a dashed lane line is found in any continuous image sub-sequence. Forthe images, the distances from the camera location to the terminal pointare calculated respectively. The vehicle speed is just derived from thedistance difference dividing by the time difference for the images.

[0024] As the described monocular computer vision technique for aidingroad vehicle driving, the left/right location of the vehicle in a roadlane is computed from the ratio of two distances which are from themidpoint of the image lower border to the intersection points of theextended image lower border and the two lines of the current lane.

[0025] As the described monocular computer vision technique for aidingroad vehicle driving to acquire the driving situation, the situationincludes the distance to a front obstacle or vehicle, the speed of thevehicle, and the left/right location of the vehicle in a road lane.

DETAILED EXPLANATION FIGURES

[0026]FIG. 1. The diagram of the invention vision system. (a) Side view.(b) Front view.

[0027]FIG. 2. The left-handed coordinate system and the cameracoordinate system used in the proposal. (a) The left-handed coordinatesystem. (b) The camera coordinate system.

[0028]FIG. 3. The image center point and a vanishing point.

[0029]FIG. 4. The side view of the relationship between the camera andthe road plane.

[0030]FIG. 5. The horizontal lines in an image.

[0031]FIG. 6. The side view of the relationship between the camera andimage horizontal lines.

[0032]FIG. 7. The front view for indicating the right/left location ofthe vehicle in the current lane.

[0033]FIG. 8. The top view of a camera coordinate system in a lane.

COMPONENTS

[0034] 11: vehicle 12: camera 13: road

[0035] [Explanation]

[0036] The present invention installs an image capture device on thefront of a vehicle as an artificial eye to capture a sequence ofroad/scene images for detecting the distance to a front obstacle orvehicle, estimating the speed of the vehicle, determining the left/rightlocation of the vehicle in a road lane, and providing sound and/or lightto warn the driver to aid the driver for safety.

[0037]FIG. 1 is the equipment diagram of the present invention. Avehicle 11 navigates on a road 13. A camera 12 is fixed near the centerof the windshield and the upper border of the windshield; the camerafaces to the road in front of the vehicle.

[0038] The theories and the formulas for the present inventiontechniques are detailedly described as follows:

[0039] A. Detecting the Distance to a Front Obstacle or Vehicle

[0040] We use the left-handed coordinate system to define the cameracoordinate system and the world coordinate system as shown in FIG. 2.The rotation angle for the coordinate system is measured clockwise. Theoptical axis of the camera is defined as the z-axis of the cameracoordinate system; the image is just the xy-plane of the cameracoordinate system. The world coordinate system is just used for helpingexplanation of the proposed theory and we only care its orientation;thus only the direction of the coordinate system is defined. Assume theroad being a plane. The world coordinate system is defined by the roadplane and the lane lines of the current lane. Let the road plane be theXZ-plane of the world coordinate system and one lane line be the Z-axisof the world coordinate system.

[0041] Let (x_(o), y_(o)) be the coordinates of the image center and(x_(v), y_(v)) be the coordinates of the vanishing point of the currentlane as shown in FIG. 3. According to the theory of vanishing point, the3-dimensional direction of the lane lines with respect to the cameracoordinate system is (x_(v), y_(v), q), where q is the focal length ofthe camera lens. If the vanishing point is not located on the centralvertical line of the image, the camera must have a yaw rotation to theworld coordinate system. The contained angle φ′ of Z-axis of the worldcoordinate system (i.e., the lane line) and x-axis of the cameracoordinate system can be calculated by the formula $\begin{matrix}{{\cos \quad \varphi^{\prime}} = {\frac{\begin{bmatrix}1 & 0 & 0\end{bmatrix} \cdot \begin{bmatrix}x_{v} & y_{v} & q\end{bmatrix}}{\sqrt{x_{v}^{2} + y_{v}^{2} + q^{2}}}.}} & (1)\end{matrix}$

[0042] The acute angle φ of the lane line and the yz-plane (i.e., theyaw angle of the camera coordinate system) is then calculated as$\begin{matrix}{\varphi = {{90{^\circ}} - {{\cos^{- 1}\left( \frac{xv}{\sqrt{x_{v}^{2} + y_{v}^{2} + q^{2}}} \right)}.}}} & (2)\end{matrix}$

[0043] The positive φ means that the camera coordinate system turns leftand the negative φ means to turn right. The component of the lane lineon the yz-plane is [0 Y_(v) q], Thus the acute angle of the lane lineand the camera optical axis (i.e., z-axis) on the yz-plane is θ,$\begin{matrix}{{\cos \quad \theta} = {\left. \frac{\begin{bmatrix}0 & y_{v} & q\end{bmatrix} \cdot \begin{bmatrix}0 & 0 & 1\end{bmatrix}}{\sqrt{y_{v}^{2} + q^{2}}}\Rightarrow\theta \right. = {{\cos^{- 1}\left( \frac{q}{\sqrt{y_{v}^{2} + q^{2}}} \right)}.}}} & (3)\end{matrix}$

[0044] From Eqs.(2) and (3), we can find the pitch angle θ and yaw angleφ of the lane lines with respect to the camera coordinate system. Thereare something in the formulas should be cared: the units of thecoordinates [x_(v), Y_(v)] and q must be the same; the pixels in imagesmay be not square pixels; thus the image should be calibrated before itis used for analysis.

[0045] The location under the camera and on the road is called thecamera location. Assume the distance from the camera center to thecamera location (i.e., the height of the camera) being h as shown inFIG. 4. If the camera location to the intersection point of the opticalaxis and the road plane is d, then by the relation of the righttriangle, we have $\begin{matrix}{\frac{h}{d} = {\left. {\tan \quad \theta}\Rightarrow d \right. = {h\quad \cot \quad {\theta.}}}} & (4)\end{matrix}$

[0046] Now we want to compute the distance from the camera location tothe road position at which a horizontal line in the image indicates;examples are shown in FIG. 5.

[0047] Assume that a horizontal line is located below the image centerwith vertical distance a₁ as shown in FIG. 6. The road position of thehorizontal line to the camera location is d₁, the view angle withrespect to the optical axis is δ₁; that is the view angle with respectto the road plane is θ₁,

θ₁−θ=δ₁.  (5)

[0048] According to the relationship of a right triangle, δ₁ can becomputed for the formula $\begin{matrix}{\frac{a_{1}}{q} = {\left. {\tan \quad \delta_{1}}\Rightarrow\delta_{1} \right. = {{\tan^{- 1}\left( \frac{a_{1}}{q} \right)}.}}} & (6)\end{matrix}$

[0049] Besides, from FIG. 6, we have

d₁=h cotθ₁.  (7)

[0050] If we substitute Eqs.(5) and (6) into Eq.(7), we have$\begin{matrix}{d_{1} = {{h\quad \cot \quad \theta_{1}} = {{h\quad {\cot \left( {\theta + \delta_{1}} \right)}} = {h\quad \cot \quad {\left( {\theta + {\tan^{- 1}\left( \frac{a_{1}}{q} \right)}} \right).}}}}} & (8)\end{matrix}$

[0051] In other words, the distance from the camera location to a roadposition at which a horizontal line in the image indicates can becomputed from known camera height h, focal length q of the camera lens,the view angle θ of the optical axis with respect to the road plane, andthe distance a₁ from the image center to the horizontal line. If thehorizontal line indicates the intersection of the rear wheel of a frontvehicle and the road plane, then we just find the distance to the frontvehicle.

[0052] We can also derive other formulas to compute the distance to afront vehicle. The derivation is started from Eq.(6) $\begin{matrix}{\frac{a_{1}}{q} = {\left. {\tan \quad \delta_{1}}\Rightarrow{\cot \quad \delta_{1}} \right. = {\left. \frac{q}{a_{1}}\Rightarrow{\cot \left( {\theta_{1} - \theta} \right)} \right. = {\frac{q}{a_{1}}.}}}} & (9)\end{matrix}$

[0053] From Eqs.(4) and (7), we have $\begin{matrix}\begin{matrix}{{d - d_{1}} = \quad {{h\left( {{\cot \quad \theta} - {\cot \quad \theta_{1}}} \right)} = {h\quad \frac{{\cot \quad {\theta cot\theta}_{1}} + 1}{\cot \left( {\theta_{1} - \theta} \right)}}}} \\{\left. \Rightarrow{d - d_{1}} \right. = \quad {{h\quad \frac{{\cot \quad {\theta cot}\quad \theta_{1}} + 1}{\frac{q}{a_{1}}}} = {\frac{a_{1}h}{q}\left( {{\cot \quad {\theta cot}\quad \theta_{1}} + 1} \right)}}} \\{\left. \Rightarrow d_{1} \right. = \quad {d - {\frac{a_{1}h}{q}{\left( {{\cot \quad \theta \quad \cot \quad \theta_{1}} + 1} \right).}}}}\end{matrix} & (10)\end{matrix}$

[0054] That is, the distance from the camera location to a road positionat which a horizontal line in the image indicates can be computed fromknown camera height h, focal length q of the camera lens, the view angleθ of the optical axis with respect to the road plane, the distance a₁from the image center to the horizontal line, the pitch angle θ of thecamera, and the distance d from the camera location to the intersectionpoint of the optical axis and the road plane.

[0055] With the same principle, a horizontal line is located above theimage center with vertical distance a₂ as shown in FIG. 6 and the roadposition of the horizontal line to the camera location is d₂, then theview angles have the relation

θ−θ₂=δ₂.  (11)

[0056] According to the relationship of the right triangle, view angleδ₂ can be calculated by $\begin{matrix}{\frac{a_{2}}{q} = {\left. {\tan \quad \delta_{2}}\Rightarrow\delta_{2} \right. = {{\tan^{- 1}\left( \frac{a_{2}}{q} \right)}.}}} & (12)\end{matrix}$

[0057] Moreover, we have

d₂=h cot θ₂.  (13)

[0058] If we substitute Eqs.(11) and (12) into Eq.(13), we have$\begin{matrix}{d_{2} = {{h\quad \cot \quad \theta_{2}} = {{h\quad {\cot \left( {\theta - \delta_{2}} \right)}} = {h\quad \cot \quad {\left( {\theta - {\tan^{- 1}\left( \frac{a_{2}}{q} \right)}} \right).}}}}} & (14)\end{matrix}$

[0059] With other derivation from Eq.(12), we have $\begin{matrix}{\frac{a_{2}}{q} = {\left. {\tan \quad \delta_{2}}\Rightarrow{\cot \quad \delta_{2}} \right. = {\left. \frac{q}{a_{2}}\Rightarrow{\cot \quad \left( {\theta - \theta_{2}} \right)} \right. = {\frac{q}{a_{2}}.}}}} & (15)\end{matrix}$

[0060] From Eqs.(4) and (13), we have $\begin{matrix}{{d_{2} - d} = {{h\left( {{\cot \quad \theta_{2}} - {\cot \quad \theta}} \right)} = {\left. {h\quad \frac{{\cot \quad {\theta cot}\quad \theta_{2}} + 1}{\cot \left( {\theta - \theta_{2}} \right)}}\Rightarrow{d_{2} - d} \right. = {{h\quad \frac{{\cot \quad {\theta cot}\quad \theta_{2}} + 1}{\frac{q}{a_{2}}}} = {\left. {\frac{a_{2}h}{q}\left( {{\cot \quad {\theta cot}\quad \theta_{2}} + 1} \right)}\Rightarrow d_{2} \right. = {d + {\frac{a_{2}h}{q}{\left( {{\cot \quad {\theta cot}\quad \theta_{2}} + 1} \right).}}}}}}}} & (16)\end{matrix}$

[0061] The process steps based on the proposed theory are described asfollows:

[0062] i. from an image to find a horizontal line to represent the touchlocation of the rear wheel of a front vehicle in the current lane andjudge the line is located above or below the image center;

[0063] ii. extract the lines of the current lane and find the vanishingpoint (x_(v), y_(v));

[0064] iii. from the vanishing point and the focal length q of thecamera lens to find the pitch angle θ and yaw angle φ of the lane withrespect to the camera coordinate system;

[0065] iv. from the pitch angle θ and the known camera height h to findthe distance d using Eq.(4);

[0066] v. find the distance a₁ (or a₂) from image center to thehorizontal line, then calculate the front-vehicle distance d₁ (or d₂)using Eq.(8) (or Eq.(14)).

[0067] B. Estimating the Vehicle Speed

[0068] We can use the above “detecting the distance to a front obstacleor vehicle” method to detect a terminal point of a dashed lane line ofthe current lane in a continuous image sub-sequence as one example shownin FIG. 5. For the images, the distances from the camera location to theterminal point are calculated respectively. The vehicle speed is justderived from the distance difference dividing by the time difference forthe images.

[0069] C. Determining the Left/Right Location of the Vehicle in a RoadLane

[0070] If the optical axis of the camera is on the YZ-plane of the worldcoordinate system, then the vanishing point of the current lane mustlocate on the vertical central line of the image. The factor isindependent to the left/right shift of the camera. In other words, onlythe camera coordinate system having a yaw rotation with respect to theworld coordinate system can result in the factor that the vanishingpoint is not located on the vertical central line of the image.

[0071] Now we use an image as shown in FIG. 7 and a top-view diagram asshown in FIG. 8 to describe the relationship of the yaw rotation of thecamera coordinate system and the lane lines. If c is the midpoint of theimage lower border shown in FIG. 7; c is also shown in the current lanein the top-view diagram. In the image, we draw a line to link point cand the vanishing point; the drawn line is just the vehicle locationline shown in FIG. 8. From the viewpoint of the 3-dimensional geometricmeanings, the line is parallel to the lane lines. In the image, we cancompute a distance ratio on any horizontal line located below the imagecenter. We track a horizontal line from the vehicle location line toboth lane lines to get two distances. The ratio of these two distancesis invariant to the location of the horizontal line and the ratio is thesame as the ratio of point c to both lane lines, L/R as shown in FIG. 7;moreover, the ratio is also the same as the ratio l/r shown in FIG. 8.Ratio l/r is just the distance ratio of the camera to the lane lines;ratio l/r is independent to the yaw rotation of the camera coordinatesystem. From the ratio, we can obtain the right/left location of thevehicle in the current lane.

[0072] If we know the vehicle width, we can judge whether the vehiclehas deviated from the current lane and driven into the neighboring laneor has not. Assume that the lane width is R_(w), the vehicle width isV_(w), and the camera is fixed at the center of the windshield. If theratio l/r is less than V_(w)/(2R_(w)-V_(w)) or greater than(2R_(w)-V_(w))/V_(w), then the vehicle has deviated from the currentlane.

SUMMARY OF THE INVENTION

[0073] We here summarize the above presentation of our invention. Thepresent invention is a monocular computer vision technique to aid roadvehicle driving for safety. The technique uses a camera which isinstalled in a vehicle and faces to the road in front of the vehicle tocapture a sequence of road/scene images to acquire the distance to afront obstacle or vehicle, the speed of the vehicle, and the left/rightlocation of the vehicle in a road lane. If a danger situation is arisen,the sound and/or light alarm will arise to warn the driver for safety.The present invention utilizes monocular computer vision technique toreduce the cost and provide multiple safety functions. The presenttechnique is an advanced, practical, and novel invention. The abovedescription may not include all parts of the invention, the fully claimof the invention is then submitted as follows.

What is claimed is:
 1. A monocular computer vision method to aid roadvehicle driving for safety, a camera being installed in a vehicle facingto a road in front of said vehicle, said method comprising steps of: i.using said camera to capture a sequence of road/scene images; ii.processing said sequence of road/scene images; iii. analyzing a drivingsituation from said processing results.
 2. A method according to claim 1wherein said camera is a digital camera or camcorder.
 3. A methodaccording to claim 1 wherein said vehicle is a car, bus, or truck.
 4. Amethod according to claim 1 wherein said camera is fixed near a centerof a windshield of said vehicle and an upper border of said windshield;said camera faces to said road in front of said vehicle; a contents ofsaid sequence of road/scene images are said road and scene in front ofsaid vehicle.
 5. A method according to claim 1 wherein said drivingsituation includes: a distance to a front obstacle or vehicle, a speedof said vehicle, and a left/right location of said vehicle in a roadlane.
 6. A method according to claim 5 wherein said distance to a frontobstacle or vehicle is determined from steps of: i. from said sequenceof road/scene images to extract two lines on both side of said road laneand then find an intersection point of said two lines; said intersectionpoint is just a vanishing point of said two lines; ii. from saidvanishing point and a known focal length of a lens of said camera tofind a pitch and yaw angles of said two lines with respect to a cameracoordinate system; iii. from said pitch angle and a height of saidcamera to calculate a distance from a location of said camera to a pointwhich is said intersection point of an optical axis of said camera and aroad plane; iv. from said images to find horizontal lines that indicatessaid intersection point of rear wheels of front vehicles and said road,select a nearest horizontal line that overlaps said road lane, and judgethat said horizontal line is located above or below an image center; andv. from said known camera focal length, camera height, pitch angle, anda vertical distance from said horizontal line to said image center tofind said distance from said camera location to said rear wheel of saidfront vehicle or obstacle.
 7. A method according to claim 6 wherein saidvehicle speed is computed based on a determination of said distance to afront vehicle; at first, a terminal point of a dashed lane line is foundin said sequence of road/scene images; for said images, said distancesfrom said camera location to said terminal point are calculatedrespectively; said vehicle speed is just derived from said distancedifference dividing by a time difference for said images.
 8. A methodaccording to claim 6 wherein said left/right location of said vehicle ina road lane is computed from a ratio of two distances which are from amidpoint of an image lower border to said intersection points of anextended image lower border and two lines of said road lane.